A Graph-Based Reinforcement Learning Method with Converged State Exploration and Exploitation

In any classical value-based reinforcement learning method, an agent, despite of its continuous interactions with the environment, is yet unable to quickly generate a complete and independent description of the entire environment, leaving the learning method to struggle with a difficult dilemma of choosing between the two tasks, namely exploration and exploitation. This problem becomes more pronounced when the agent has to deal with a dynamic environment, of which the configuration and/or parameters are constantly changing. In this paper, this problem is approached by first mapping a reinforcement learning scheme to a directed graph, and the set that contains all the states already explored shall continue to be exploited in the context of such a graph. We have proved that the two tasks of exploration and exploitation eventually converge in the decision-making process, and thus, there is no need to face the exploration vs. exploitation tradeoff as all the existing reinforcement learning methods do. Rather this observation indicates that a reinforcement learning scheme is essentially the same as searching for the shortest path in a dynamic environment, which is readily tackled by a modified Floyd-Warshall algorithm as proposed in the paper. The experimental results have confirmed that the proposed graph-based reinforcement learning algorithm has significantly higher performance than both standard Q-learning algorithm and improved Q-learning algorithm in solving mazes, rendering it an algorithm of choice in applications involving dynamic environments

Effect of Heat Treatment on Microstructure and Hardness of a Worn Rail Repaired Using Laser Powder Deposition

The frequent replacement of worn rails on tracks brings an immense economic burden on the railroad industry, and also causes significant interruptions to railroad operation. Restoration of worn rails via laser powder deposition (LPD) can considerably reduce the associated maintenance costs. This study was focused on the use of LPD to repair the worn profile of a standard U.S. rail. The microstructure of the 304L stainless steel deposits with a minimum hardness of 85 HRB was composed of austenite, δ-ferrite, and sigma. Micropores were dispersed throughout the deposit, and microcracks were found at the rail-deposition interface. The pearlitic rail substrate showed a moderate hardness of 94 HRB. The fine-grain, pearlitic-ferritic heat affected zone had the maximum hardness of 96 HRB, which was still below the minimum required hardness of 97 HRB for a typical rail. To increase the hardness to or above 97 HRB and to mitigate the microstructural defects, the as-repaired rail went through a heat treatment process. The average hardness of the as-treated rail was increased significantly, i.e., to 103 HRB. Besides, the porous and coarse-grain deposition materials were transformed into an impermeable and fine-grain microstructure. However, heat treatment intensified the microcracks at the rail-deposition interface and also led to the formation of martensite and augmentation of the micropores in the parent rail. Isolation of the base rail during heat treatment and preheating were suggested as solutions for the problematic results. The LPD process ultimately was found to be a promising technique for repairing rails

Feasibility Study of a Campus-Based Bikesharing Program at UNLV

Bikesharing systems have been deployed worldwide as a transportation demand management strategy to encourage active modes and reduce single-occupant vehicle travel. These systems have been deployed at universities, both as part of a city program or as a stand-alone system, to serve for trips to work, as well as trips on campus. The Regional Transportation Commission of Southern Nevada (RTCSNV) has built a public bikesharing system in downtown Las Vegas, approximately five miles from the University of Nevada, Las Vegas (UNLV). This study analyzes the feasibility of a campus-based bikesharing program at UNLV. Through a review of the literature, survey of UNLV students and staff, and field observations and analysis of potential bikeshare station locations, the authors determined that a bikesharing program is feasible at UNLV

The Impacts of Emergency Vehicle Signal Preemption on Urban Traffic Speed

We used GPS data from paratransit vehicles to evaluate the impact of emergency vehicles on urban traffic speeds. The results indicate that speed variance is significantly higher during emergency preemption and the mean speeds of traffic flowing in the same direction as the emergency vehicle and on crossing streets are lower during preemption than during normal conditions. Regression results indicate that traffic on major arterials and traffic in the opposite direction of the emergency vehicle tend to have higher speed during signal preemption. Signal preemption during peak periods and duration of preemption had a significant negative impact on traffic speeds. Also, the transition time has a negative impact on traffic speeds. The authors recommend further research on how to optimize (minimize) the preemption duration as well as transition time. Also, the impact of median type and number of lanes should be evaluated

Detecting changes in freeway traffic states using the CUSUM algorithm

The process of detecting a change in freeway traffic states should fully utilize the available information about the traffic processes before and after a state change as well as the state change occurrence distribution. The literature review revealed that only parts of this information are used by existing state change detection algorithms. In this thesis, state change detection algorithms were developed by applying the CUSUM algorithm where varying levels of a priori information on changes in traffic states can be incorporated. To apply the CUSUM algorithm, the properties of traffic processes before and after state changes and the distributions of state change occurrence were first investigated. The results indicated that it may not be realistic to apply the original CUSUM algorithm, in which full information about changes is assumed to be known. Thus, four algorithms were designed by varying the amount of prior information integrated in the formulation. These algorithms and some relevant state-of-the-art algorithms were evaluated under different simulation conditions. A practical evaluation with field data was also performed. The four versions of the CUSUM algorithm are promising for use in practice due to their desirable performance. These algorithms can be applied to other transportation problems such as determining uniform pavement sections

Comparison of Methods for Defining Geographical Connectivity for Variables of Trip Generation Models

The four-step procedure for travel demand modeling is ubiquitously used in the United States and elsewhere due to institutional and financial requirements. Trip generation analysis, which is the first step, is based on traffic analysis zones which are geographical units. Therefore, the zonal trip generation totals are observations measured at different geographical locations. Spatial distribution of the observations limits the methods that can be applied in analyzing the data and influences the final conclusions that can be reached. Most past efforts of incorporating spatial effects in trip generation models have been using contiguity of the zones as the major criterion for defining spatial relationship of the observations. In this study, we tested for the presence of spatial auto-correlation in both trip attraction and trip production variables. We found significant spatial auto-correlation in trip attraction variables but insignificant spatial auto-correlation in trip production variables for the data collected from the Las Vegas valley. We evaluated four alternative methods for defining geographical connectivity: (1) contiguity, (2) separation, (3) combined contiguity and separation, and (4) economic linkage (accessibility measure). Comparison of the trip attraction model indicated that the model estimated using geographical connectivity matrix with elements defined by separation (distance between centroids of the zones) was the best fitted. A further comparison indicated that this model outperformed the one without spatial variables by minimizing deviation of the modeled trips from observed trips. We recommend using separation between the zones to define spatial connectivity as opposed to contiguity only. Contiguity only excludes zones that may have an impact on the observation under consideration

Freeway Incident Likelihood Prediction and Response Decision-Making

This research project consisted of two parts. The first part developed a set of real-time incident likelihood prediction models. The second part developed a freeway incident response decision-making methodology based on sequential hypothesis testing methods. The freeway incident likelihoods predicted by the real-time prediction models act as prior probabilities for the freeway incident response decision-making system. The products of this research project will be incorporated in the Advanced Traffic Management System that is being implemented on the Borman Expressway, a 16-mile segment of I-80 in northwest Indiana. The decision-making system can be used by traffic management personnel to assist in responding to various freeway incidents in a near optimal manner to minimize traffic delays and reduce the number of secondary incidents

An Information-Based Time Sequential Approach to Online Incident Duration Prediction

Online prediction of incident duration (i.e., remaining incident duration) is becoming more important as more intelligent transportation system deployments in the United States focus on improving the operations of transportation management centers. This study proposed a time sequential procedure where an incident management process is divided into stages according to the specific information available. For each stage, a hazard-based duration regression model with different variables representing the available information was developed. By calibrating these models, the parameters of probability distributions assumed for incident duration and the coefficients of the variables were jointly estimated. Based on the estimated parameters and coefficients, remaining incident duration can be predicted online using the truncated median of incident duration. This study concluded that the accuracy of the prediction of incident duration increases as more information is incorporated into the developed models. The prediction based on the truncated median is more accurate than that based on the truncated mean because the probability distribution of incident duration has a long tail. The proposed procedure provides flexibility for implementation in the real world

Safety impact of access management techniques at signalized intersections

Signalized intersections next to each other spatially on the same arterial share some unobservable information such as traffic flow and roadway characteristics, thus this study is to investigate the impact of access management techniques on crash counts at signalized intersections where the unobservable information is considered by developing panel data crash count data models based on crash data from 300 signalized intersections in Southern Nevada. Panel data random-effect model was selected to take into account the unobserved factors for each unique arterial. It was showed that Negative Binomial regression models were better able to reflect the dispersion in the crash data. Therefore, the random-effect negative binomial model (RENB) was applied to investigate the relationship between crash occurrence and access management techniques. The results showed that five variables significantly affected the safety at signalized intersections. The average length of corner clearance had negative impact on intersection crash occurrence while the total traffic flow in all directions, land use types, the number of lanes for minor streets and posted speed limit on minor streets were positively related to crashes at signalized intersections